System and method for ordering haptic effects

ABSTRACT

A signal associated with multiple haptic effects is received, each haptic effect from the multiple haptic effects being associated with a time slot from multiple time slots. Each haptic effect from the multiple haptic effects is associated with an effect slot from multiple effect slots at least partially based on the time slot associated with that haptic effect. An output signal is sent for each effect slot from the multiple effect slots, when the associated haptic effect is scheduled for its time slot.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 14/033,756filed on Sep. 23, 2013 (herein incorporated by reference), which is adivisional of U.S. patent application Ser. No. 13/479,969 filed on May24, 2012 (herein incorporated by reference), which is a divisional ofU.S. patent application Ser. No. 12/842,626, filed on Jul. 23, 2010(herein incorporated by reference), which is a continuation applicationof U.S. patent application Ser. No. 10/983,229, filed on Nov. 8, 2004(herein incorporated by reference), which claims priority of U.S.Provisional Application No. 60/587,904 filed Jul. 15, 2004 (hereinincorporated by reference).

BACKGROUND

The invention relates generally to haptic feedback devices. Morespecifically, the invention relates to systems and methods for orderinghaptic effects.

Devices that provide tactile feedback, such as haptic feedback, haveenjoyed increased popularity in recent years. These devices are used ina variety of different applications. For example, devices providinghaptic feedback are popular in various computer gaming applications,where the haptic feedback enhances the overall gaming experience of auser. In addition to gaming applications, haptic feedback devices alsohave been used in a variety of other computer-application contexts. Forexample, haptic-enabled controllers, such as mouse devices, can beconfigured to provide haptic feedback to a user while the user interactswith an operating system (OS), or other application.

Similarly, tactile feedback has been incorporated in various virtualreality applications to enhance the overall experience of a user. Forexample, haptic feedback can be incorporated in a virtual environment toprovide a more realistic interactive simulation experience. Surgerysimulation, for example, is a virtual reality environment where hapticfeedback has been used to provide a user with a more realisticexperience and, hence, a more educational experience.

Tactile feedback has also been increasingly incorporated in portableelectronic devices, such as cellular telephones, personal digitalassistants (PDAs), portable gaming devices, and a variety of otherportable electronic devices. For example, some portable gamingapplications are capable of vibrating in a manner similar to controldevices (e.g., joysticks, etc.) used with larger-scale gaming systemsthat are configured to provide haptic feedback. Additionally, devicessuch as cellular telephones and PDAs are capable of providing variousalerts to users by way of vibrations. For example, a cellular telephonecan alert a user to an incoming telephone call by vibrating. Similarly,a PDA can alert a user to a scheduled calendar item or provide a userwith a reminder for a “to do” list item or calendar appointment.

Generally, vibrations output by standard portable electronic devices,such as PDAs and cellular telephones, are simple vibrations, whichoperate as binary vibrators that are either on or off. That is, thevibration capability of those devices is generally limited to afull-power vibration (a “fully on” state), or a rest state (a “fullyoff”). Thus, generally speaking, there is little variation in themagnitude of vibrations that can be provided by such devices.

Existing devices, however, are rather rudimentary and do not providesophisticated tactile feedback. Accordingly, it would be desirable toprovide more sophisticated vibrations, which can convey additionalinformation (e.g., by way of a sophisticated series of effects), andwhich can provide a user with an enhanced tactile experience beyond whatis possible using devices that are currently available.

SUMMARY

An embodiment of the invention provides an apparatus that includes anordering component and an output component. The ordering component isconfigured to associate each basis haptic effect from multiple basishaptic effects with a time slot from multiple time slots. The outputcomponent is configured to associate each basis haptic effect from themultiple basis haptic effects with an effect slot from multiple effectslots. The output component is also configured to cause each basishaptic effect from the multiple basis haptic effects to be output duringthe time slot associated with that haptic effect.

Another embodiment of the invention provides a method that receives asignal associated with multiple haptic effects. Each haptic effect fromthe multiple haptic effects is associated with a time slot from multipletime slots. The method also associates each haptic effect from themultiple haptic effects with an effect slot from multiple effect slotsat least partially based on the time slot associated with that hapticeffect. The method also sends an output signal for each effect slot fromthe multiple effect slots, when the associated haptic effect isscheduled for its time slot. The method can be implemented by aprocessor device, such as a computer, using a processor-readable mediumcomprising computer code representing instruction configured to cause aprocessor to implement the method.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a processor system, according to anembodiment of the invention.

FIG. 2 is a block diagram of a system configured to order hapticeffects, according to an embodiment of the invention.

FIG. 3 is a block diagram of a system configured to order hapticeffects, according to another embodiment of the invention.

FIG. 4 is a block diagram of a system configured to order hapticeffects, according to another embodiment of the invention.

FIG. 5 is a block diagram of an output component, according to anembodiment of the invention.

FIG. 6 is a block diagram of a file structure, according to anembodiment of the invention.

FIG. 7 is a block diagram of a timeline-effect file structure, accordingto an embodiment of the invention.

FIG. 8 is a block diagram of an effect-storage-block file structure,according to an embodiment of the invention.

FIG. 9 is a block diagram of an effect-name block file structure,according to an embodiment of the invention.

FIG. 10 is a block diagram of a timeline-effect definition, according toan embodiment of the invention.

FIG. 11 is a timeline diagram illustrating the execution of multiple,basis haptic effects, according to an embodiment of the invention.

FIG. 12 is block diagram of memory addresses for multiple, basis hapticeffects, according to an embodiment of the invention.

FIG. 13 is a diagram of a computer window of an application formodifying a periodic haptic effect, according to an embodiment of theinvention.

FIG. 14 is a diagram of a computer window of an application for orderingmultiple, basis haptic effects to form a timeline haptic effect,according to an embodiment of the invention.

FIG. 15 is a diagram of a computer window of an application for creatinga haptic effect from a sound file, according to an embodiment of theinvention.

FIG. 16 is a diagram of a computer window of an application for editinga frequency mapping of a haptic effect from a sound file, according toan embodiment of the invention.

DETAILED DESCRIPTION

Systems and methods for ordering haptic effects are described. Morespecifically, an embodiment of the invention is described in the contextof a system and method configured to output a series of “basis,” orfundamental, haptic effects based on a time slot associated with each ofthe basis haptic effects. Basis haptic effects are output in response tobasis haptic-effect signals. The basis haptic-effect signals are eachassociated with one of several effect slots based on the time slotassociated with that basis haptic effect. The series of basis hapticeffects output according to one or more embodiments of the invention iscapable of conveying additional information to users that has not beenpossible using known devices. Moreover, the series of basis hapticeffects output according to one or more embodiments of the invention isalso capable of providing a user with an enhanced experience while usinga device, such as a portable electronic device.

As used herein, the term “basis haptic effect” refers to an elementalhaptic effect that can be used as a component or “building block” ofmore complex haptic effects. A “basis haptic-effect signal” is a signalthat causes a basis haptic effect to be output (e.g., via a hapticdevice). Basis haptic effects are also sometimes referred to herein as“haptic effects” for the sake of simplicity.

According to one or more embodiments of the invention, basis hapticeffects can include many types of effects, such as periodic hapticeffects and magnitude-sweep haptic effects. These basis haptic effectscan be used as components or “building blocks,” for example, to form atimeline haptic effect. Each of the components of the timeline effectcan be ordered in such a way to provide a specific, desired tactileoutput to a user. The ordering of these components (e.g., basis hapticeffects) can occur, for example, based on a time slot associatedtherewith.

An event-driven scheme, similar to schemes used for MIDI files, or thelike, can be used to output timeline haptic effects. Timeline hapticeffects can be created by way of an interactive software-authoring tool,which allows a developer to specify the nature, order, and timing ofbasis haptic effects used to create a timeline haptic effect.Additionally, or alternatively, timeline haptic effects can be createdby automatically converting information from files having anevent-driven scheme (e.g., MIDI files, etc.).

According to one or more embodiments of the invention, an architecture(which can include software, firmware, hardware, or any combinationthereof) is provided. The architecture can include an ordering componentconfigured to associate each basis haptic effect from multiple basishaptic effects with a time slot during which that basis haptic effect isto be output. This can be accomplished, for example, based on one ormore control signals received by the ordering component (e.g., from aninterface component). The ordering component can, for example, providean ordered haptic-effect signal (e.g., a timeline haptic-effect signal)to a driver, which is configured to cause all basis haptic effectsassociated with the timeline haptic-effect signal to be output via ahaptic device. The output component is configured to associate eachbasis haptic effect with an effect slot from multiple effect slots. Theoutput component is also configured to cause each basis haptic effect tobe output during its associated time slot.

Advantageously, one or more embodiments of the invention allow hapticeffects to be output based on information obtained from, converted from,or otherwise associated with a corresponding output (e.g., as defined bya MIDI file, etc.). For example, in the case of event-driven files, suchas MIDI files, haptic effects can be output based on the notes,channels, or any combination thereof, defined in the MIDI file. Thisallows haptic effects to be output synchronously with such event-drivenfiles (e.g., synchronized with the beat of music created by a MIDIfile). Following this example, a different haptic effect can be definedfor each MIDI instrument and/or channel, or combination thereof, suchthat a great variety of haptic effects is available (e.g., each hapticeffect can correspond to one of over 200 MIDI instruments). Moreover,haptic effects can be commanded to correspond to parameters of MIDIfiles (e.g., corresponding to notes in length, speed, strength,duration, etc.), or can be based on such parameters, but tailored asdesired, according to one or more other embodiments of the invention.Editing such parameters can be carried out prior to, during, or afterselection of corresponding features in the MIDI file.

FIG. 1 is a block diagram of a processor system, according to anembodiment of the invention. The processor system 110 illustrated inFIG. 1 can be, for example, a commercially available personal computer,portable electronic device, or a less complex computing or processingdevice (e.g., a device that is dedicated to performing one or morespecific tasks). For example, the processor system can be a cellulartelephone, a PDA, a portable gaming system, or the like. Alternatively,the processor system 110 can be a terminal dedicated to providing aninteractive virtual reality environment, such as a gaming system, or thelike. Although each component of the processor system 110 is shown asbeing a single component in FIG. 1, the processor system 110 can includemultiple numbers of any components illustrated in FIG. 1. Additionally,multiple components of the processor system 110 can be combined as asingle component.

The processor system 110 includes a processor 112, which according toone or more embodiments of the invention, can be a commerciallyavailable microprocessor capable of performing general processingoperations. Alternatively, the processor 112 can be anapplication-specific integrated circuit (ASIC) or a combination ofASICs, which is designed to achieve one or more specific functions, orenable one or more specific devices or applications. In yet anotheralternative, the processor 112 can be an analog or digital circuit, or acombination of multiple circuits.

Alternatively, the processor 112 can optionally include one or moreindividual sub-processors or coprocessors. For example, the processorcan include a graphics coprocessor that is capable of renderinggraphics, a math coprocessor that is capable of efficiently performingcomplex calculations, a controller that is capable of controlling one ormore devices, a sensor interface that is capable of receiving sensoryinput from one or more sensing devices, and so forth.

The processor system 110 can also include a memory component 114. Asshown in FIG. 1, the memory component 114 can include one or more typesof memory. For example, the memory component 114 can include a read onlymemory (ROM) component 114 a and a random access memory (RAM) component114 b. The memory component 114 can also include other types of memorynot illustrated in FIG. 1 that are suitable for storing data in a formretrievable by the processor 112. For example, electronicallyprogrammable read only memory (EPROM), erasable electricallyprogrammable read only memory (EEPROM), flash memory, as well as othersuitable forms of memory can be included within the memory component114. The processor system 110 can also include a variety of othercomponents, depending upon the desired functionality of the processorsystem 110. The processor 112 is in communication with the memorycomponent 114, and can store data in the memory component 114 orretrieve data previously stored in the memory component 114.

The processor system 110 can also include a haptic device 116, which iscapable of providing a variety of tactile feedback. For example, thehaptic device 116 can be configured to output basis haptic effects, suchas periodic effects, magnitude-sweep effects, or timeline hapticeffects, each of which is described in greater detail below. Accordingto one or more embodiments of the invention, the haptic device 116 caninclude one or more force-applying mechanisms, which are capable ofproviding tactile force to a user of the processor system 110 (e.g., viathe housing of the processor system 110). This force can be transmitted,for example, in the form of vibrational movement caused by the hapticdevice 116 (e.g., caused by a rotating mass, a piezo-electric device, orother vibrating actuator), or in the form of resistive force caused bythe haptic device 116.

The processor system 110 can also, according to one or more embodimentsof the invention, include a sensor 118 that is capable of receivinginput from a user, the haptic device 116, or is otherwise capable ofsensing one or more physical parameters. For example, according to oneor more embodiments of the invention, a sensor 118 can be configured tomeasure speed, intensity, acceleration, or other parameters associatedwith a haptic effect output by the haptic device 116. Similarly, thesensor 118 can be configured to sense environmental or ambientconditions of the processor system's surroundings. The sensor 118 caninterface and communicate with the processor 112 by way of a sensorinterface (not shown) within the processor 112.

The processor system 110 can also include a controller 120, which canoptionally be internal to the processor 112, or external thereto, asshown in FIG. 1. The controller 120 can be configured to control thehaptic device 116 when the processor 112 is not directly controlling thehaptic device 116. Similarly, the controller 120 can control the memory114 and/or the sensor 118, as well as devices external to the processorsystem 110 by way of an input/output (I/O) component 124 (describedbelow).

The various components of the processor system 110 can communicate withone another via a bus 122, which is capable of carrying instructionsfrom the processor 112 and/or the controller 120 to other components,and which is capable of carrying data between the various components ofthe processor system 110. Additionally, signals received via the sensor118 can be communicated to the processor 112 or the controller 120 byway of the bus 122. Data retrieved from or written to memory 114 iscarried by the bus 122, as are instructions to the haptic device 116.Instructions to the haptic device 116 can be provided in the form ofhaptic-effect signals (e.g., basis haptic-effect signals), for example,which can be provided by the processor 112, the controller 120, ordevices external to the processor system 110.

The components of the processor system 110 can communicate with devicesexternal to the processor system 110 by way of an input/output (I/O)component 124 (accessed via the bus 122). According to one or moreembodiments of the invention, the I/O component 124 can include avariety of suitable communication interfaces. For example, the I/Ocomponent 124 can include, for example, wireless connections, such asinfrared ports, optical ports, Bluetooth wireless ports, wireless LANports, or the like. Additionally, the I/O component 124 can include,wired connections, such as standard serial ports, parallel ports,universal serial bus (USB) ports, S-video ports, large area network(LAN) ports, small computer system interface (SCSI) ports, and so forth.

FIG. 2 is a block diagram of a system configured to order hapticeffects, according to an embodiment of the invention. The variouscomponents of the system 200 illustrated in FIG. 2 are configured tocause the haptic device 116 to output a series of basis haptic effects(e.g., arranged as a timeline haptic effect). According to one or moreembodiments of the invention, software components of the system shown inFIG. 2 are illustrated with solid lines, while hardware components areillustrated with dashed lines. It will be appreciated, however, that atleast some components of the system 200 implemented using software canbe implemented using hardware of firmware. Moreover, any combination ofsoftware, hardware, and firmware can be used for any components shown inFIG. 2, depending upon the desired functionality of the system 200.

The system 200 of FIG. 2 includes an interface component 202, which canbe configured to communicate with a client application external to thesystem 200. The interface component 202 can serve as a connection pointto the system 200 for a client application external to the system 200.For example, according to one or more embodiments of the invention, theinterface component 202 can include an application-programming interface(API), which is configured to provide access to the capabilities of thesystem 200 for an application program outside of that system 200.According to one or more embodiments of the invention, the interfacecomponent 202 can be configured to run on the same processor as anapplication that accesses the system 200 via the interface component202. For example, both the interface component 202 and the accessingapplication program can run on a processor similar to the processor 112of the processor system 110 described above (shown in FIG. 1).Alternatively, the interface component 202 and the application programaccessing the system 200 via the interface component can run onseparate, interconnected processors.

According to one or more embodiments of the invention, an applicationexternal to the system 200 (e.g., a client application), is able toperform a variety of functions with regard to haptic effects to beoutput by the haptic device by using the interface component 202 toaccess the functionality of the system 200 shown in FIG. 2. For example,an application can create, modify, or delete effect signals, which areconfigured to cause corresponding effects, using the interface component202. Additionally, an application can start and stop effects on thehaptic device 116 via the interface component 202. According to one ormore embodiments of the invention, the interface component 202 can allowa client application to modify haptic effects in real-time, or “on thefly,” while the effect is being implemented. Such an ability can befacilitated, for example, by way of feedback using a sensor, such as thesensor 118 (shown in FIG. 1), or by way of optional feedback to variouscomponents within the system (e.g., shown using dashed lines in FIG. 2).

The interface component 202 is in communication with a communicationbridge 204, which is configured to provide access between the interfacecomponent 202 and the remaining components of the system 200. Forexample, according to one or more embodiments of the invention where theinterface component 202 resides on a first processor and the remainingcomponents of the system 200 shown in FIG. 2 reside on a secondprocessor, the communication bridge 204 provides a connection betweenthe first and second processors. For example, the communication bridge204 can use a known protocol for communicating between multipleprocessors, such as a universal serial bus (USB) interface, or othersuitable protocol.

According to one or more embodiments of the invention, the communicationbridge 204 is optional and can be omitted, if desired. For example, theinterface component 202 and the remaining components of the system 200shown in FIG. 2 (usually excluding the communication bridge 204) can runon a single processor. In such an implementation, the communicationbridge 204 may not be necessary. For example, the interface component202 can be configured to communicate directly with the communicationbridge 204. Although a hardware communication bridge 204 may not beneeded between two software components on a single processor, a softwarecommunication bridge 204 can be used in such a situation, if desired.Accordingly, the communication bridge 204 is entirely optional, and canbe modified to include a software communication bridge 204, dependingupon the desired implementation of the system 200.

A driver 206 coordinates communication between various components of thesystem 200, including an output component 208, a resource manager 210,an ordering component 212, and an output register 214. The driver 206queries these components for action items, and coordinatescommunications and data transfer between the components according toexisting action items. The driver 206 also communicates with theinterface component 202 (e.g., via the communication bridge 204, orotherwise), whereby it can receive and handle requests from, orcommunicate data to client applications external to the system 200.

According to one or more embodiments of the invention, the driver 206coordinates various time-sensitive operations of the various components(e.g., the output component 208, the resource manager 210, and theordering component 212) of the system 200 with which it communicates. Tothis end, the driver 206 can query various components of the system 200at relatively short time intervals, such as one-millisecond (ms) or0.5-ms intervals. Of course, the length of time intervals the driver 206uses to query components can be modified according to variousapplication requirements for which the system 200 is to be used.

The driver 206 communicates with an output component 208, which usesvarious control algorithms to cause the haptic device 116 to outputspecified haptic effects during their associated time slots. The outputcomponent 208 outputs basis haptic-effect signals, which are configuredto cause haptic effects (e.g., periodic effects, magnitude sweepeffects, etc.) to be output by haptic device 116. The output component208 can modify basis haptic-effect signals, for example, by varyingspecific parameters of the haptic effects to be output by the hapticdevice 116. For example, the output component 208 can use controlalgorithms to specify or vary intensity, periodicity, ramp-up time,ramp-down time, and other parameters of the haptic effects that to beoutput by the haptic device 116.

The output component 208 can be configured to output multiple hapticeffect signals simultaneously. For example, according to one or moreembodiments of the invention, the output component 208 can output up tofour basis haptic-effect signals, simultaneously, each of which isconfigured to cause a different haptic effect to be output by the hapticdevice 116. These multiple basis haptic-effect signals can occupymultiple corresponding “effect slots” within the output component 208,which are used to differentiate the various signals that can be outputsimultaneously by the output component 208. Thus, the output component208 associates each haptic effect with an effect slot (e.g., byassociating a corresponding basis haptic-effect signal with an effectslot) during the time slots associated with the haptic effects.According to one or more embodiments of the invention where the outputcomponent 208 is capable of outputting four basis haptic-effect signalssimultaneously, the output component 208 can include four effect slots.It will be appreciated, however, that the number of effect slots withinthe output component 208 can be varied based on a number of factors.

A resource manager 210 communicates with the output component 208, andacts as a supporting component for the output component 208.Specifically, the resource manager 210 manages the output component 208and haptic-effect signals to be output by the output component 208 bycontrolling the timing with which a specific haptic effect is to beoutput. In response to control signals from the resource manager 210,the output component 208 generates a basis haptic-effect signal (e.g.,which can be communicated to the driver 206 in the form of a pulse-widthmodulation signal) configured to cause the haptic effect to be output,as directed by the resource manager 210. The driver 206 can then passthis basis haptic-effect signal, as described below, to other componentsof the system 200 to cause the haptic device 116 to output the desiredhaptic effects.

According to an embodiment where the output component 208 has multipleeffect slots, the resource manager 210 can determine which slot shouldbe filled with which effect signals. This can occur, for example, basedon a time slot associated with the haptic effect. Generally, theresource manager 210 will cause an effect signal for a new effect to beplaced into the next available slot in the output component 208. Whenall of the effect slots within the output component 208 are active(i.e., outputting a haptic-effect signal), and none are available, theresource manager 210 determines which of the effect signals currentlybeing output should remain, and which effect signals within the outputcomponent 208, if any, should be replaced by a new haptic-effect signal.

When a new haptic-effect signal is desired and all of the effect slotsof the output component 208 are already active, the resource manager 210can examine the effects in each of the effect slots of the outputcomponent 208 to determine the amount of time remaining for each effectsignal being output. The resource manager 210 can, additionally oralternatively, determine a priority between multiple effects. Based atleast partially on one or more such determinations, the resource manager210 can determine whether a haptic-effect signal currently being output(i.e., an active haptic-effect signal), should be removed to allow a newhaptic-effect signal to be inserted in an effect slot (i.e., to beoutput by the output component 208), or whether the active haptic-effectsignal should remain, and have priority over, a new haptic-effectsignal. Priority can be determined, for example, on the basis of thetime remaining for the active effect signal, or on the basis of apredetermined parameter, such as a priority level previously assigned toan effect signal, or other parameters. According to one or moreembodiments of the invention, the resource manager 210 can simply removean effect being executed by the output component 208 that has the leasttime remaining so that execution of a new effect by the output component208 is facilitated.

The ordering component 212 performs as a sequencer, providing a timelinefor the driver 206 to use in outputting the basis haptic-effect signalsproduced by the output component 208. Specifically, the orderingcomponent 212 provides an order and timing for each of the basishaptic-effect signals to be executed by the output component 208. Forexample, the ordering component 212 can interpret a series ofinstructions, which can be presented in the form of a timeline-effectfile or a timeline haptic effect signal, and issue commands regardingwhich basis haptic-effect signals are to be output at certain times, andin which order those signals are to be output. This can occur, forexample, based on a time slot associated with each haptic effect to beoutput using a basis haptic-effect signal, which can be defined in afile or other signal (e.g., received from outside the system 200). It isusing such commands from the ordering component 212 that the resourcemanager 210 can determine which basis effect-signals are to be output atspecific times by the output component 208 to cause basis haptic effectsto be output by the haptic device 116 at desired times. According to oneor more embodiments of the invention, the ordering component 212 issuescommands at least partially based on a timeline-effect file receivedfrom a client application outside of the system 200 via the interfacecomponent.

According to one or more embodiments of the invention, the driver 206,the output component 208, the resource manager 210 and the orderingcomponent 212 can run on a single processor, which is different from theprocessor upon which the interface component 202 runs. As mentionedabove, these two processors can be interconnected by way of a physical,hardware communication bridge 204. According to one or more embodimentsof the invention, each of these processors can be similar to theprocessor 112 (shown in FIG. 1), and can be within a single processorsystem 110 (shown in FIG. 1) or among multiple processor systems 110.Alternatively, however, the interface component 202 can run on the sameprocessor as the driver 206, the output component 208, the resourcemanager 210, and the ordering component 212. Depending upon the specificimplementation, the communication bridge 204 can be implemented in theform of a software component rather than a hardware component, tocommunicate between the interface component 202 and the driver 206.

An output register 214 can be used by the system 200 to receive theordered, basis haptic-effect signals, which can be represented, forexample, in the form of a timeline haptic-effect signal. According toone or more embodiments of the invention, the output register 214 can bea hardware device, such as a pulse width modulation (PWM) outputregister configured to receive and/or output a PWM signal, or othersuitable output register device.

The ordered, basis haptic-effect signals are communicated from theoutput register 214 to the device driver 216. The device driver 216 canbe a hardware device, according to one or more embodiments of theinvention. For example, the device driver 216 can include electroniccomponents and circuitry used to supply the haptic device 116 with therequired electric current to cause the ordered haptic-effect signalsspecified in the output register 214 to output corresponding hapticeffects on the haptic device 116. For example, the current provided bythe device driver 216 to the haptic device 116 can have varyingmagnitudes of positive and negative current. Additionally, oralternatively, the current can be in the form of periodic signals withvarying periods and/or phases, which correspond to the characteristicsof the series of ordered haptic effects stored in the output register214.

The configuration of the system 200 shown in FIG. 2 can be modified inmany ways according to the desired performance of the system 200. Forexample, various components shown in FIG. 2 can be omitted or combined.Moreover, additional components can be added to the system 200 shown inFIG. 2. Two examples of alternate configurations of the system 200 shownin FIG. 2 are illustrated in FIG. 3 and FIG. 4. These alternateconfigurations, however, are merely examples of possibleimplementations, and are not intended to exhaustively demonstrate allpossible configurations.

FIG. 3 is a block diagram of a system configured to order hapticeffects, according to another embodiment of the invention. The system300 shown in FIG. 3 is similar to the system 200 of FIG. 2, but does notinclude an ordering component 212 (shown in FIG. 2). Rather, anyordering of haptic-effect signals, or assigning of haptic-effect signalsto effect slots can be accomplished using the resource manager 210,which also manages the resources of the output component 208, asdiscussed above. Thus, the resource manager 210 can manage both timeslots and effect slots associated with haptic effects to be output,according to one or more embodiments of the invention.

FIG. 4 is a block diagram of a system configured to order hapticeffects, according to another embodiment of the invention. The system400 shown in FIG. 4 is similar to the system 200 of FIG. 2 with somedifferences. For example, the system 400 of FIG. 4 is configured tooperate in a “multi-mode” configuration. More specifically, multiplemodes of output can be processed and output simultaneously using thesystem 400 of FIG. 4.

For example, multiple processing components 402 a, 402 b (sometimesreferred to collectively or individually as processing component(s) 402)can receive signals (e.g., from the interface component 202, via thecommunication bridge 204, etc.) and can order haptic-effect signalsconfigured to cause haptic effects to be output. The orderedhaptic-effect signals can be output to the output register 214, thedevice driver 216, and/or the haptic device 116. Although only twoprocessing components 402 are shown, more can be included to provideadditional modes, if desired. Additionally, each of the processingcomponents 402 is optional, and can, instead, be implemented as acollection of other components. Each of the processing components 402can access the functionalities of its individual components, which caninclude, for example, a resource manager 210, an ordering component 212,and/or an output component 208, among other components.

The multi-mode system 400 shown in FIG. 4 uses a first-mode outputcomponent 208 a included in the first processing component 402 a toprocess and output information associated with a first mode, and asecond-mode output component 208 b included in the second processingcomponent 402 b. According to one or more embodiments of the invention,the first-mode output component 208 a can be configured to cause hapticeffects to be output in the manner described above. The output caused bythe first-mode output component 208 a can be synchronous with the outputcaused by the second-mode output component 208 b. For example, accordingto one or more embodiments of the invention, the second-mode outputcomponent 208 b can be configured to cause non-haptic output, such asaudio output (e.g., from MIDI files, WAV files, etc.), video output, orother suitable output. According to one or more embodiments of theinvention, output caused by the second-mode output component 208 b canbe created according to an event-driven set of instructions (e.g., aMIDI file), and the haptic effects caused by the first-mode outputcomponent 208 a can be created based on the event-driven set ofinstructions either manually, or by automatic conversion, as will bedescribed below in greater detail. By using such a multi-mode system400, synchronicity between the multiple modes produced by the system 400can be achieved.

FIG. 5 is a block diagram of an output component, according to anembodiment of the invention. The output component 208 and its componentsare illustrated in detail in FIG. 5, where the output component 208 isshown in communication with the driver 206 and the resource manager 210,in the same configuration shown in FIG. 2. The output component 208includes a haptic output component 502, by which the driver 206 and theresource manager 210 communicate with the output component 208. Thehaptic output component 502 can make use of various algorithms withinthe output component 208 to package various data packets that are to becommunicated to the driver 206 or the resource manager 210 from othercomponents within the output component 208.

The output component 208 can include one or more controllers 504 a, 504b, 504 c (collectively, individually, or as a subset referred to hereinas controller(s) 504). Each of the controllers 504 within the outputcomponent 208 communicate with an effect generator 506, which is incommunication with the haptic output component 502. The controllers 504can make use of an algorithm within the output component 208 to cause abasis haptic-effect signal to be output. The effect generator 506 canmonitor the controllers 504 and track timing between the variouscontrollers 504. The effect generator 506 can also communicate effecttiming (e.g., the time remaining for an effect, etc.) to the hapticoutput component 502, which can communicate this information to theresource manager 210 to allow the resource manager 210 to manage thebasis haptic-effect signals output by the output component 208.Additionally, basis haptic-effect signals provided by the controllers504 to the effect generator cause the effect generator 506 to output avalue selected to cause the haptic device 116 (shown in FIGS. 1 and 2)to output a basis haptic effect. For example, according to one or moreembodiments of the invention, the effect generator 506 can output a PWMvalue, that can be interpreted by the output register 214 (shown in FIG.2), after being received from the driver 206. The output register 214can use this value to cause the device driver 216 to drive the hapticdevice 116.

The output component 208 also includes an effect manager 508 that can beused by the haptic output component 502 to process communicationsreceived from the driver 206. For example, according to one or moreembodiments, packets from the driver 206 containing encoded effectparameters (e.g., as part of a basis haptic-effect signal) can beprocessed by the effect manager 508. This can include, for example,receiving and parsing the received communications, and extracting valuesencoded within those communications, which can then be converted touseful formats, if necessary. The extracted and/or converted values canbe stored in one or more internal memory locations to be used by theeffect generator 506 in generating the appropriate signal to cause acorresponding commanded haptic effect to be output. According to one ormore embodiments of the invention, the effect manager 508 decodestime-related values that are received in an encoded format from thedriver. Additionally, the effect manager 508 can handle or modifyinformation that affects behavior of various components of the outputcomponent 208 and the system 200 (shown in FIG. 2) in which it isemployed. For example, the effect manager 508 can modify variousparameters used to determine force values to be output by a hapticdevice 116 (shown in FIG. 2). This can be accomplished, for example,using a linearization table that is used to convert haptic-effect signalparameters to force values. Using such a linearization table, the effectmanager 508 can change the amount of force calculated by the effectgenerator 506 to be output for corresponding haptic-signal values.

The output component 208 also includes an INIT/close engine 510, whichinitializes all structures or variables used by the controllers 504, theeffect generator 506, and the effect manager 508. The INIT/close engine510 also ensures that the driver 206 provides a valid memory locationfor writing force values associated with the haptic-effect signals(e.g., as calculated by the effect generator 506). Additionally, theINIT/close engine 510 ensures that all memory at termination time (e.g.,after all haptic-effect signals have been output) is properly reset.

FIG. 6 is a block diagram of a file structure, according to anembodiment of the invention. More specifically, in FIG. 6, a hierarchy600 of a file structure used to order multiple, basis haptic effects tocreate timeline haptic-effect signals is illustrated. A file structuredaccording to the hierarchy 600 shown in FIG. 6 can be used, for example,by the ordering component 212 to determine the order and timing of basishaptic effects to be executed by the haptic device 116. It will beappreciated, however, that the hierarchy 600 shown in FIG. 6 is only onepossible implementation. According to one or more embodiments of theinvention, the hierarchy 600 shown in FIG. 6 can be varied, or replacedby another hierarchy suitable for the desired function of theimplementation in which is it used. Files having the hierarchy 600 shownin FIG. 6 can be created or modified by a client application, which canaccess the functionality of the system 200 shown in FIG. 2 by way of theinterface component 202 of that figure.

As shown in the hierarchy 600 of FIG. 6, a timeline-effect file can becreated and/or saved in extensible markup language (XML). According toone or more other embodiments of the invention, however, thetimeline-effect file can be created and/or stored in another format,such as other markup languages, binary, or other suitable formats,depending upon on the desired implementation or design constraints. Thetimeline-effect file defines two separate classes, “effects” and“multimedias.”

The effects class includes a timeline-effect subclass and a basis-effectsubclass. Two types of basis effects are defined: periodic and magnitudesweep (also referred to as “mag sweep” effects). Periodic effects areeffects that provide periodic sensations, such as periodic vibrations,or similar effects. Periodic effects can be driven using a periodicdrive signal (e.g., sinusoidal signal, square-wave signal, etc.).Periodic effects can also be driven using constant signals that causeperiodic motion (e.g., a direct-current drive signal that causes a motorto spin, etc.). Magnitude sweep effects are effects that provide acertain sensation magnitude or group of magnitudes (e.g., a constantforce, a constant resistance, etc.).

Two events of timeline effects are defined: a launch event and a repeatevent. A launch event specifies a basis haptic-effect signal and a timeat which the haptic effect corresponding to the basis haptic-effectsignal is to be started, or launched. A repeat event specifies a starttime and a duration during which events are to be repeated a specifiednumber of times.

Another class of effects, called “multimedias” can optionally includethe basis-effect subclass. This class of effects is used, for example,to synchronize effects, such as basis haptic effects, with multimediafiles (e.g., MIDI files, WAV files, MP3 files, etc.). The “multimedias”class causes the basis haptic effects to be output simultaneously andsynchronously with the corresponding multimedia file. Thus, by using anauthoring tool according to one or more embodiments of the invention, a“multimedias” class can be designed such that when music from a mediafile is output by a device, one or more basis haptic effects can beoutput by a haptic device (e.g., integral to the device outputting themusic) synchronously with the multimedia file. This can be accomplished,for example, using a multi-mode system 400 (shown in FIG. 4). Accordingto one or more embodiments of the invention, the timeline-effect filesstructured according to the hierarchy 600 shown in FIG. 6 areillustrated in greater detail in FIGS. 7, 8, and 9.

FIG. 7 is a block diagram of a timeline-effect file structure, accordingto an embodiment of the invention. A timeline-effect file 500 can beconfigured for use by a client application (e.g., an applicationaccessing the system 200 of FIG. 2, and application of an embeddedsystem, etc.) to create a timeline-effect signal, or an ordered seriesof basis haptic-effect signals. The timeline-effect file 500 shown inFIG. 7 can include a header block 702, which contains a definition ofthe timeline-effect signal to cause a desired, ordered series or basishaptic effects to be output by the haptic device 116. The header block702 can also include information regarding the format version and sizeof the timeline-effect file 500, the number (and order of) basis hapticeffects to be generated by the timeline-effect signal, and the locationof other blocks within the timeline-effect file 500, such as theeffect-storage block 704 and the optional effect-name block 706.

In addition to the header block 702, the timeline-effect file 500 canalso include an effect-storage block 704 and an optional effect-nameblock 706. The effect-storage block 704 contains information regardingeach effect (e.g., basis haptic effect or timeline, haptic effect, etc.)defined in the timeline-effect file 500. For example, according to oneor more embodiments of the invention, the effect-storage block 704 cancontain definitions of each effect within the timeline-effect file 500.The optional effect-name block 706 can be used to store the names ofeach of the effects within the timeline-effect file 500. Table 1 belowshows a header that can be used as the header block 702 of thetimeline-effect file 500, according to an embodiment of the invention.

TABLE 1 Header Block 702 Byte Contents Meaning 0 0x01 Major Versionnumber (e.g., 1 for file format 1.x). 1 0x00 Minor Version number (e.g.,0 for IVT file format x.0). 2 EFFECTCOUNT_15_8 Number of effects, bits15 to 8. 3 EFFECTCOUNT_7_0 Number of effects, bits 7 to 0. 4ESBSIZE_15_8 Bits 15 to 8 of the Effect-Storage Block size in bytes. 5ESBSIZE_7_0 Bits 7 to 0 of the Effect-Storage Block size in bytes. 6ENBSIZE_15_8 Bits 15 to 8 of the Effect-Name Block size in bytes. 7ENBSIZE_7_0 Bits 7 to 0 of the Effect-Name Block size in bytes.

FIG. 8 is a block diagram of an effect-storage-block file structure,according to an embodiment of the invention. As shown in FIG. 8, theeffect-storage block 704 includes an effect-storage-offset sub-block 802and an effect-storage-data sub-block 804. The effect-storage-offsetsub-block 802 defines where each effect begins in the timeline-effectfile 500. Specifically, according to one or more embodiments of theinvention, the effect-storage-offset sub-block 802 is an array ofoffsets, each of which corresponds to a separate effect defined withinthe timeline-effect file 500. Each offset within the array can includetwo bytes of information that specify where the definition for eacheffect begins in the effect-storage-data sub-block 804. According to oneor more embodiments of the invention, the size of theeffect-storage-offset sub-block 802 includes a number of bytes that istwice the number of total effects defined in the timeline-effect file500.

The effect-storage-data sub-block 804 stores the definitions of each ofthe effects defined in the timeline-effect file 500, including, forexample, basis haptic effects and timeline effects. Each type of effectstored in the effect-storage-data sub-block 804 can be stored in amanner that renders it easily identifiable. For example, according toone or more embodiments of the invention, the least significant“nibble,” or four bits, of data of each effect can contain a specifiedvalue depending upon the type of effect it is. For example, the leastsignificant nibble of the basis haptic effects can have a value of 0x0,while the least significant nibble of the timeline effects can have 0xF.

FIG. 9 is a block diagram of an effect-name-block file structure,according to an embodiment of the invention. As shown in FIG. 9, theoptional effect-name block 706 includes an effect-name-offset sub-block902, and an effect-name-data sub-block 904. The effect-name-offsetsub-block 902 can include an array of offsets, each of which correspondsto an effect defined within the timeline-effect file 500. As with theeffect-storage-offset sub-block 802, the size of each offset within theeffect-name-offset sub-block 902 can be two bytes, such that the totalsize of the effect-name-offset sub-block 902 includes a number of bytesthat is twice the number of total effects defined in the timeline-effectfile 500. Each offset in the effect-name-offset sub-block 902 specifieswhere the name of the corresponding to the effect is located in theeffect-name-data sub-block 904. The effect-name-data sub-block 904stores the effect names of each effect defined by the timeline-effectfile 500.

FIG. 10 is a block diagram of a timeline-effect definition, according toan embodiment of the invention. Specifically, the timeline-effectdefinition 1000 of a first timeline haptic effect (“Timeline1”) isillustrated in FIG. 10. On the left-hand side of the timeline-effectdefinition 1000 are several circles that each correspond to a uniquebasis haptic effect and are to be included in the first timeline hapticeffect (e.g., including a first and second periodic effect, a constanteffect, and a spring effect, etc.). On the right-hand side of thetimeline-effect definition 1000, the basis haptic effects are shown inan ordered fashion, as events. Each event indicates the basis hapticeffect and the launch time when that effect is to begin within thetimeline effect. In other words, timeline-effect definition 1000 isconfigured to output a series of ordered, basis haptic-effect signals,each of which corresponds to the basis haptic effect defined in thetimeline-effect definition 1000, and each of which begins at a launchtime indicated in the timeline-effect definition 1000.

As shown in FIG. 10, the various ordered series of basis haptic-effectsignals, shown on the right-hand side of the timeline-effect definition1000 as events, can specify additional information. For example, certainoverride values can be specified for the various basis haptic-effectsignals (e.g., a duration override, a magnitude override, a periodoverride, etc.), which set a value different from a default value forone or more parameters of the basis haptic-effect signal. Additionally,priorities can be set for each of the basis haptic-effect signals to aidthe resource manager 210 (shown in FIG. 2) in determining which effectsignals have priority over others to occupy effect slots within theoutput component 208.

FIG. 11 is a timeline diagram illustrating the execution of multiple,basis haptic effects, according to an embodiment of the invention. Theexample of a timeline 1100 shown in FIG. 11 illustrates the manner inwhich the system 200 (shown in FIG. 2) causes the output component 208(shown in FIG. 2) to implement a timeline effect (e.g., the orderedbasis haptic effects) specified in a timeline-effect definition 1000(shown in FIG. 10). First, an application (e.g., a client applicationaccessing the system 200 of FIG. 2) calls for the first timeline effectTimeline 1 to be started. In response, the ordering component 210 (shownin FIG. 2) creates or instantiates each of the basis haptic-effectsignals defined in the timeline-effect definition (e.g., a first andsecond periodic effect, a constant effect, and a spring effect, etc.).Once each basis haptic-effect signal is created or instantiated, theordering component proceeds to call each basis haptic-effect signal inthe order and at the time specified in the timeline-effect definition1000 (shown in FIG. 10), implementing any override values for each basishaptic-effect signal.

FIG. 12 is block diagram of memory addresses for multiple, basis hapticeffects, according to an embodiment of the invention. The compactbasis-effect-storage block 1200 shown in FIG. 12 can be used toimplement a compact storage schema for storing multiple, basis hapticeffects where space is a premium (e.g., in embedded applications, etc.).In the compact basis-effect-storage block 1200, only two types ofmessages, or instructions, are stored: a SetPeriodic message and aSetPeriodicModifier message. Each of these messages can be used to storedefinitions of basis haptic-effect signals (e.g., periodic haptic-effectsignals, constant haptic-effect signals, etc.). When stored as part of atimeline effect signal, each of these messages can be padded to 8 bytesin length. The SetPeriodic message can call for the execution of a basishaptic effect, by way of a corresponding, specified basis haptic-effectsignal. The SetPeriodicModifier message operates to modify theSetPeriodic message (e.g., providing an envelope for the basishaptic-effect signal, etc.).

The compact basis-effect-storage block 1200 includes multiple basishaptic-effect signals, each of which can have lengths of either 8 bytes(e.g., an unmodified SetPeriodic message) or 16 bytes (e.g., aSetPeriodic message modified by a SetPeriodicModifier message). Thus, inthe compact basis-effect-storage block 1200, multiple basishaptic-effect signals can be ordered in a predetermined order, andstored using an index (e.g., 0, 1, and 2, for the effect signals“Effect0,” “Effect1,” and “Effect 2,” respectively, each shown on theleft-hand side of FIG. 12) and a relative memory address location (e.g.,0x0000, 0x0008, 0x0010, etc.). The index and relative memory addresslocation can be used by the output component 208 (shown in FIG. 2) tocall the basis haptic-effect signals in order, and to cause thecorresponding basis haptic-effect signals to be executed in that order.

FIG. 13 is a diagram of a computer window of an application formodifying a periodic haptic effect, according to an embodiment of theinvention. Similar windows can be provided for modifying other types ofbasis haptic effects. The modification computer window 1300 shown inFIG. 13 is an interface that can form, for example, a part of a clientapplication configured to create or modify haptic effects. Such a clientapplication can, for example, access the system 200 (shown in FIG. 2) byway of the interface component 202. According to one or more embodimentsof the invention, the modification computer window 1300 (or the othercomputer windows described below) can form part of a stand-alonecomputer application program. Alternatively, according to one or moreother embodiments of the invention, the modification computer window1300 (or the other computer windows described below) can form part of orinterface with an existing computer application program (e.g., can be aplug-in, etc.). For example, the modification computer window 1300 (orthe other computer windows described below) can form part of systemdescribed in U.S. Pat. Nos. 5,959,613 and 6,278,439 to Rosenberg et al.,entitled, “Method and Apparatus for Shaping Force Signals for a ForceFeedback Device,” each of which is incorporated by reference herein inits entirety.

The effect being created in the modification computer window 1300 shownin FIG. 13 is a periodic effect, and is represented by the periodiceffect icon on the left-hand side of the window 1300. That periodiceffect fades in magnitude from a maximum magnitude value over time,becomes constant for a period of time, and subsequently fades to zeromagnitude. Various parameters can be changed by way of the modificationcomputer window 1300 shown in FIG. 13, such as the magnitude, style,attack time, attack level, fade time, fade level, duration, speed, andperiod of the haptic effect. It is by way of such a modificationcomputer window 1300 that individual desired basis haptic effects can betailored, such that a tailored corresponding basis haptic-effect signalis created. Once the basis corresponding haptic-effect signal iscreated, it can be placed within a timeline haptic-effect signal, to beimplemented at a desired time and in a desired order relative to otherbasis haptic-effect signals within the timeline haptic-effect signal.

FIG. 14 is a diagram of a computer window of an application for orderingmultiple, basis haptic effects to form a timeline haptic effect,according to an embodiment of the invention. In FIG. 14, a timelineeffect computer window 1400 is shown allowing multiple, basis hapticeffects to be organized into a timeline haptic effect. For example, onthe left-hand side, several icons corresponding to different types ofeffects are shown, including a periodic haptic effect, a magnitude-sweephaptic effect, and a timeline haptic effect. These effects can bemodified in a window similar to the modification computer window 1300(shown in FIG. 13) discussed above. The MIDI file entitled “fbn” is alsoshown on the left-hand side of the timeline effect computer window 1400.The MIDI file is also graphically represented in the main portion of thetimeline effect computer window 1400 along with the various effects tobe output synchronously with the MIDI's audio output. More specifically,in the main portion of the timeline effect computer window 1400,graphical representations of notes and channels are shown at varioustimes along the timeline.

Each of the icons shown on the left-hand side of the timeline effectcomputer window 1400 can be interacted with via the graphical userinterface (GUI) in which the timeline effect computer window ispresented. For example, a user can “drag-and-drop” an icon into thetimeline haptic effect being constructed in the main portion of thewindow, or to exchange MIDI files to be output synchronously with sucheffects. Each effect that forms part of the timeline haptic effect canbe highlighted, and parameters of that effect can be adjusted. Forexample, the periodic effect that is highlighted has override fields onthe right for specifying start time (which can also be specified oraltered in drag-and-drop fashion), magnitude, and duration. Otheroverride values, which are not set for the highlighted periodic effectcan also be used with other effects (e.g., speed, period, etc.).Additional detail of each effect can also be viewed by “double clicking”on the effect or its icon (e.g., detail similar to the detail shown inFIG. 13).

FIG. 15 is a diagram of a computer window of an application for creatinga haptic effect from a sound file, according to an embodiment of theinvention. According to one or more embodiments of the invention, asound file (e.g., a MIDI file) can be automatically converted to one ormore timeline-effect files, such as the timeline-effect file 500 (shownin FIG. 7) using predefined conversion parameters. First, basishaptic-effect signals can be developed for each type of MIDI instrumentto be converted. A library storing all such signals can be developed andupdated to contain all known MIDI instruments, if desired. According toone or more embodiments of the invention, these basis haptic-effectsignals can be named with the same name as the corresponding MIDIinstruments, which are provided in the MIDI specification (e.g.,acoustic grand piano, bright acoustic piano, electric grand piano,honky-tonk piano, etc.).

Automatic conversion from a sound file (e.g., a MIDI file) to a timelinehaptic effect file can be accomplished in a variety of ways. Accordingto one or more embodiments of the invention, a sound file can beanalyzed for suitability of automatic conversion. If it is determinedthat the file is not suitable for automatic conversion, then an alertcan be generated, and the file can be rejected. If the sound file issuitable to be automatically converted, then the MIDI conversioncomputer window 1500 can be presented to a user, each of the identifiedinstruments being displayed with the corresponding channel and asuggested frequency map.

The MIDI conversion computer window 1500 shown in FIG. 15 illustratessome options that can be selected to facilitate conversion from a MIDIfile to a timeline haptic effect file. For example, the MIDI channelname and corresponding MIDI instruments can be identified. Ifautomatically converting the MIDI file to a timeline haptic effect isdesired, the “import” option can be selected. Options can also beselected for automatic conversion of MIDI durations to haptic effectdurations, MIDI note velocity to haptic effect magnitude, MIDI notenumber to haptic effect frequency (which can be adjusted using thefrequency map described below), and so forth. It should be recognizedthat additional conversions can be implemented depending upon thedesired conversion to be performed by one or more embodiments of theinvention.

MIDI files can be converted in several ways. For example, eachinstrument can be converted to haptic effects (e.g., in a haptic-effectfile). Similarly, each channel of the MIDI file can be converted tohaptic effects. Additionally, any instrument-channel combination definedin a MIDI file can be converted to haptic effects (e.g., using anautomatic conversion technique). In many instances, conversion of allnotes defined for a particular channel in a MIDI file can be selectedfor automatic conversion to haptic-effect signals. In some instances,conversion of notes for each instrument, or even each note within achannel, may be desirable. For example, on MIDI channel 10, eachpercussion instrument corresponds to an individual note within thatchannel. Thus, to accurately convert certain percussion instruments tohaptic effects, individual notes within channel 10 should be mapped orotherwise converted to haptic effects (e.g., by creating appropriatehaptic-effect signals).

The conversion from MIDI files to files defining haptic effects can usethe various events of the MIDI file. For example, MIDI files generallyhave “note-on-events” and “note-off-events” indicating the beginning orend of a MIDI note, respectively. Both note-on-events andnote-off-events specify a MIDI channel number, a note number, and a notevelocity. MIDI files can also have other events, such as a“program-change event” that indicates changes in instrumentation for agiving MIDI channels. For example, a program-change event can specifythat 30 seconds into a song, the instrument used on channel 1 is to bechanged from a first instrument (e.g., acoustic guitar) to a secondinstrument (e.g., electric guitar).

Conversion of MIDI information to haptic-effect information can occurusing several techniques. For example, a duration-override command canbe used to match a note-on-event with its corresponding note-off-event(i.e., the note-off-event for the same note as the note-on-event) todetermine note durations. A haptic effect can then be commanded (e.g.,using a basis haptic-effect signal or a timeline haptic-effect signal)to have the determined note duration, or the duration can be changed bymanually overriding the automatic duration to set a desired duration.Additionally, a velocity-to-magnitude command can be used to set themagnitude of an effect to be commanded (e.g., using a basishaptic-effect signal or a timeline haptic-effect signal) based on thevelocity of a MIDI note (e.g., as specified in the note-on-event thatgenerated the note with which the haptic effect is to be associated orsynchronized). Furthermore, a note-to-speed command can set thefrequency of a commanded haptic effect based on a MIDI note number(e.g., the note number for a MIDI note specified in the note-on-eventfor the note with which the haptic effect is to be associated orsynchronized).

Additionally, components of the timeline haptic effect can be createdusing default parameters, or using custom parameters specified in aconversion reference file, which can be selected using the MIDIconversion computer window 1500. The frequency map of the one or morechannels can be edited by selecting one or more “edit freq. map” buttonsshown on the right-hand side of the MIDI conversion computer window1500. Each of the options listed above can be selected on achannel-by-channel basis, an instrument-by-instrument basis, or acombination thereof.

Table 2 below shows some options for automatically converting variousMIDI messages to timeline events (e.g., events of the timeline-effectdefinition 1000 of FIG. 10).

TABLE 2 Some options for automatic conversion from MIDI to timelineevents MIDI messages Timeline events Note On LaunchPlayback Note OffDuration between a “Note On” and a “Note Off” determines the value ofduration override in LaunchPlayback event Program Change Effect IDoverride in LaunchPlayback event Bank Select Effect ID override inLaunchPlayback event Effect ID is a combination of “Bank Select” and“Program Change” All Sound Off “All Sound Off” is treated as a “NoteOff” for all pending “Note On” All Notes Off “All Notes Off” is treatedas a “Note Off” for all pending “Note On” Set Tempo “Set Tempo” MIDIevent determines TimeOffset property of a given LaunchPlayback event

FIG. 16 is a diagram of a computer window of an application for editinga frequency mapping of a haptic effect from a sound file, according toan embodiment of the invention. According to one or more embodiments ofthe invention, selecting one or more “edit freq. map” buttons shown onthe right-hand side of the MIDI conversion computer window 1500 (shownin FIG. 15) will access the frequency map computer window 1600 shown inFIG. 16. The frequency map shown in the frequency map computer window1600 plots all existing MIDI channels in a MIDI file to be converted toa timeline effect frequency. Each of the channels (shown on thehorizontal axis) can be mapped to a vibration frequency of a periodichaptic effect (shown on the vertical axis) using a translation curve1602, with which a user can interact with using the GUI within which thewindow 1600 is presented (e.g., using “click-and-drag” techniques,etc.). The translation curve 1602 defines the frequencies of a hapticdevice that correspond to the MIDI channels of the MIDI file beinganalyzed. This translation curve 1602 can be adjusted to change whicheffect frequencies will be output for sounds in a given MIDI channel. Ofcourse, other mappings (e.g., as defined using differently shapedtranslation curves, etc.) can be used for other types of haptic effectsaccording to one or more embodiments of the invention.

From the foregoing, it can be seen that systems and methods for orderinghaptic effects are discussed. Specific embodiments have been describedabove in connection with a system that is configured to order multiple,basis haptic effects to create timeline haptic effects. A file format isprovided for defining a timeline haptic effect. Additionally, variousaspects of an operating system are provided for creating, editing,modifying, and/or ordering haptic effects to create a timeline hapticeffect.

It will be appreciated, however, that embodiments of the invention canbe in other specific forms without departing from the spirit oressential characteristics thereof. For example, while some embodimentshave been described in the context of timeline haptic effects createdusing multiple ordered basis haptic effects, one can also use timelinehaptic effects, as well as other types of effects, as components tocreate other timeline effects according to the principles describedabove. Additionally, the GUI computer windows and file formats describedherein are intended as examples only, and can be varied in numerous waysto provide the same or substantially similar functionality, all of whichis considered within the scope of the invention.

What is claimed is:
 1. A computer-readable medium having instructionsstored thereon that, when executed by a processor, causes the processorto order a plurality of haptic effects, the ordering comprising:receiving a signal associated with the plurality of haptic effects, eachhaptic effect from the plurality of haptic effects being associated witha time slot from a plurality of time slots; associating each hapticeffect from the plurality of haptic effects with an effect slot from aplurality of effect slots at least partially based on the time slotassociated with that haptic effect; synchronizing a haptic effect to amultimedia file; and sending an output signal for each effect slot fromthe plurality of effect slots, when the associated haptic effect isscheduled for its time slot.
 2. The computer-readable medium of claim 1,the ordering further comprising synchronously outputting a haptic effectwith the multimedia file.
 3. The computer-readable medium of claim 1,wherein each time slot from the plurality of time slots defines a timeduring which each associated haptic effect is output.
 4. Thecomputer-readable medium of claim 1, the ordering further comprisingdefining a timeline for the plurality of time slots, the plurality ofhaptic effects being output at least partially based on the definedtimeline
 5. The computer-readable medium of claim 1, the orderingfurther comprising outputting the plurality of haptic effects.
 6. Thecomputer-readable medium of claim 1, wherein the multimedia filecomprises one of: a music instrument digital interface (MIDI) file; awaveform audio file format (WAV) file; or a MPEG-1 or MPEG-2 audio layerIII (MP3) file.
 7. A computer-implemented method for ordering aplurality of haptic effects, the method comprising: receiving a signalassociated with the plurality of haptic effects, each haptic effect fromthe plurality of haptic effects being associated with a time slot from aplurality of time slots; associating each haptic effect from theplurality of haptic effects with an effect slot from a plurality ofeffect slots at least partially based on the time slot associated withthat haptic effect; synchronizing a haptic effect to a multimedia file;and sending an output signal for each effect slot from the plurality ofeffect slots, when the associated haptic effect is scheduled for itstime slot.
 8. The method of claim 7, further comprising synchronouslyoutputting a haptic effect with the multimedia file.
 9. The method ofclaim 7, wherein each time slot from the plurality of time slots definesa time during which each associated haptic effect is output.
 10. Themethod of claim 7, further comprising defining a timeline for theplurality of time slots, the plurality of haptic effects being output atleast partially based on the defined timeline
 11. The method of claim 7,further comprising outputting the plurality of haptic effects.
 12. Themethod of claim 7, wherein the multimedia file comprises one of: a musicinstrument digital interface (MIDI) file; a waveform audio file format(WAV) file; or a MPEG-1 or MPEG-2 audio layer III (MP3) file.
 13. Anapparatus that outputs haptic effects, the apparatus comprising: aninterface component configured to receive a signal associated with theplurality of haptic effects, each haptic effect from the plurality ofhaptic effects being associated with a time slot from a plurality oftime slots; an ordering component configured to associate each hapticeffect from the plurality of haptic effects with an effect slot from aplurality of effect slots at least partially based on the time slotassociated with that haptic effect; wherein the order component isfurther configured to synchronize a haptic effect to a multimedia file;and an output component configured to send an output signal for eacheffect slot from the plurality of effect slots, when the associatedhaptic effect is scheduled for its time slot.
 14. The apparatus of claim13, wherein the output component is further configured to synchronouslyoutput a haptic effect with the multimedia file.
 15. The apparatus ofclaim 13, wherein each time slot from the plurality of time slotsdefines a time during which each associated haptic effect is output. 16.The apparatus of claim 13, wherein the ordering component is furtherconfigured to define a timeline for the plurality of time slots, theplurality of haptic effects being output at least partially based on thedefined timeline
 17. The apparatus of claim 13, wherein the outputcomponent is further configured to output the plurality of hapticeffects.
 18. The apparatus of claim 13, wherein the multimedia filecomprises one of: a music instrument digital interface (MIDI) file; awaveform audio file format (WAV) file; or a MPEG-1 or MPEG-2 audio layerIII (MP3) file.
 19. A computer-readable medium having instructionsstored thereon that, when executed by a processor, causes the processorto order a plurality of haptic effects, the ordering comprising:receiving a signal associated with the plurality of haptic effects, eachhaptic effect from the plurality of haptic effects being associated witha time slot from a plurality of time slots; associating each hapticeffect from the plurality of haptic effects with an effect slot from aplurality of effect slots at least partially based on the time slotassociated with that haptic effect; synchronizing a haptic effect to amultimedia file; and creating a timeline haptic-effect signal tosynchronize the haptic effect to an effect slot.
 20. Thecomputer-readable medium of claim 19, wherein the multimedia filecomprises one of: a music instrument digital interface (MIDI) file; awaveform audio file format (WAV) file; or a MPEG-1 or MPEG-2 audio layerIII (MP3) file.